Mixed Mode Matrix Multiplication

In modern clustering environments where the memory hierarchy has many layers (distributed memory, shared memory layer, cache, ), an important question is how to fully utilize all available resources and identify the most dominant layer in certain computation. When combining algorithms on all layers together, what would be the best method to get the best performance out of all the resources we have? Mixed mode programming model that uses thread programming on the shared memory layer and message passing programming on the distributed memory layer is a method that many researchers are using to utilize the memory resources. In this paper, we take an algorithmic approach that uses matrix multiplication as a tool to show how cache algorithms affect the performance of both shared memory and distributed memory algorithms. We show that with good underlying cache algorithm, overall performance is stable. When underlying cache algorithm is bad, superlinear speedup may occur, and increasing number of threads may also improve performance. 1. Memory Hierarchies in the modern clustering environments Figure 1 shows the memory hierarchy that exists in most nodes of modern clustering environments. Globally, many nodes are linked together by a high-speed network; inside each node there may be many processors; along with each processor memory access is either to a high speed memory unit “cache” or the low speed “main memory”. In our mixed-mode programming model we use message passing interface, MPI, for the data communication between the global nodes. Inside each MPI process we have two choices, one is to use POSIX threads for creating threads, one or many threads may belong to the MPI process mapped to the node. The other choice is again to use MPI for local processes mapped to all processors of the node. Inside each process we use different algorithms that utilize the cache. Another option for threads that was not explored in this project was to use the OpenMP standard. Based on the specific programming model, we selected several matrix multiplication algorithms on each layer and implemented them. Bova et. al., [3] determined that, “On a 100-CPU machine, using 100 MPI workers to perform a 100-component harbor simulation is inefficient due to inappropriate load balance. It would be more efficient to have 25 MPI workers create four OpenMP threads for each assigned wave component.” In our experiments, we show that even in a perfectly load balanced computation such as matrix multiplication, the overall mixed mode performance is highly affected by cache algorithms. Our testing platform is the IBM SP system at the National Energy Research Scientific Computing facility [1].